Chronos is our central lab server - primarily built to support large-scale computational work, including projects: (i) using Deep Learning/AI (ii) frequently handling large data-sets or which produce output files of significant size. Chronos currently has the following specifications:

OS: Ubuntu 22.04.5 LTS
Kernel: 6.8.0-47-generic
CPU: AMD Ryzen Threadripper PRO 5975 (32 Cores/64 Threads)
GPU: NVIDIA RTX A6000 48GB
RAM: 256GB*
HDDs: 30TB (Total)
Backup Power: Eaton 9E2000I UPS**

Chronos is optimized for multi-user workloads and we encourage you to freely use the system when needed, or as your daily work machine if you so wish. If Chronos is currently experiencing an extremely high workload (see: Section - Server Status), please consult with those currently logged in to know when resources will be freed. However, in our experience so far, there has been no need to do this.

Most importantly, Chronos was built with freedom in mind - the freedom to prototype and try new things without the fear of a monthly bill (i.e. this is not Google Cloud).

*Note: Chronos employs sophisticated memory compression techniques to significantly extend the capacity of the machine beyond 256GB of RAM. In practice, Chronos can sustain RAM usage up to ~550-650GB, before overflowing onto disk-swap.

**Note: An uninterruptible power supply (UPS) device automatically provides backup power to Chronos during our frequent, but short-lived power cuts in the faculty, and prevents loss of work. ~20-100 minutes of power is provided depending on workload. A system-wide alert is issued, to any connected terminal, upon power loss in order to provide users an opportunity to save any and all work.

Accessing Chronos

As required by the Technion’s security policy, you must be present in the faculty, on campus, or connected to the Technion VPN (VPN Setup) to access and use Chronos*.

Each user has their own, unique login credentials (username/password) and private working space within Chronos. For first-time users, your username along with a temporary password will be provided to you by an administrator.

Chronos is an OpenSSH server, meaning you can access it via any SSH-client using the following details:

Host: 192.114.18.118
Port: 22

Or directly on a SSH-enabled terminal using:

ssh username@192.114.18.118 #For example: ssh tim@192.114.18.118

For Windows users, we recommend MobaXTerm as an all-in-one SSH-client solution that will allow you to connect to Chronos but also manage/transfer files to and from the server (https://mobaxterm.mobatek.net/).

For R and Python users, access to a terminal is also provided directly through RStudio (see: Section: Using R - RStudio) and Jupyter Lab (see: Section: Using Python - JupyterLab) respectively - without the need to connect separately.

*Note: the initial VPN setup procedure can only be completed from within the Technion network (i.e. in the faculty, on campus).

Changing Passwords

To help keep Chronos secure, it is mandatory to change your temporary password after connecting for the first time. From any SSH-terminal (e.g. MobaXTerm, Rstudio, Jupyter Lab) use the following command:

passwd

And follow the instructions provided. If you forget your password, an administrator can reset it for you at anytime.

User Storage Space

Each user on Chronos has a private storage area that nobody else can access or modify. This is located at:

/media/chronos/Storage/$USER #For example: /media/chronos/Storage/tim

For ease of use, a shortcut (‘UserSpace’) to this area is present in your /home/ directory (the default directory upon logging in). In other words, you can replace the above path with:

~/UserSpace

All user data is currently stored on a dedicated 20TB HDD. While we do not enforce a per-user storage limit, we do ask users to be mindful of others and refrain from storing duplicate files or unnecessarily large data on Chronos.

Transferring & Managing Files

Files can be directly downloaded onto Chronos, or transferred from an external computer to your UserSpace (and vice versa), in a multitude of ways.

For Windows users, we recommend using MobaXTerm (https://mobaxterm.mobatek.net/) - an all-in-one SSH, X11-enabled client. MobaXTerm allows users to drag and drop files to and from your UserSpace, move or delete files, as well as open a remote web browser to directly download files onto Chronos*.

For example, after logging in to Chronos from MobaXTerm, use the following command to open a new browser session:

google-chrome & #Open a chrome instance, running on Chronos

During your first-run, please configure Chrome to automatically download files into your UserSpace, without prompt (i.e. disable “Ask where to save each file before downloading”). X11-forwarding does not fully support secondary windows, so a save dialog may not always appear.

Alternatively, files can be directly downloaded onto Chronos independently of a browser using a direct URL or through git. For example, from any connected terminal:

wget -P ~/UserSpace [URL] #For example: wget -P ~/UserSpace/Folder http://www.test.com/test.zip

git clone https://github.com/shenorrLabTRDF/TimeAx ~/UserSpace/Folder

Lastly, files can also be transferred or managed from within RStudio or JupyterLab.

Optimal transfer speeds will be achieved when connected to the Technion LAN (i.e. in the faculty/on campus) and will be considerably slower when connected to the VPN from home.

*Note: remote browser access requires the user to maintain an active connection to Chronos, which may not be suitable for extremely large downloads. Furthermore, this service requires X11 support (i.e. MobaXTerm) and will not work in all terminals.

Backups

Each UserSpace is incrementally backed up once a day, at 19:00, to our Dropbox account*.

Files stored outside of your UserSpace are not backed up. It is therefore mandatory Chronos policy that all work is stored in this area.

Dropbox limits us to low transfer rates of 10-12 files/second. Thus, in the rare cases where a UserSpace folder contains a significant number of files (especially small files, e.g. >50,000) etc., we recommend excluding this directory from backup using the DNBU (Do Not Backup) function and compressing the folder into a single file for backup instead. For example:

DNBU add ~/UserSpace/FolderToExcludeFromBackup #Exclude this folder and all contents from backup

To then compress the folder:

tar --use-compress-program='pigz -p 16' -cf ~/UserSpace/FolderToExcludeFromBackup.tar.gz ~/UserSpace/FolderFolderToExcludeFromBackup #Compress the folder, using 16 CPU cores

To remove an exclusion or check the current exclusion status of a given folder:

DNBU remove ~/UserSpace/FolderToExcludeFromBackup
DNBU check ~/UserSpace/FolderToExcludeFromBackup

*Note: Backups are mirrors i.e. if you delete a file on Chronos, it is deleted on Dropbox. However, Dropbox will retain a restorable copy of all deleted files for 6 months. Moreover, Dropbox will store versioning information for each file, allowing files to be reverted back to an older copy (as far back as 6 months) when needed. A future update to Chronos will enable a new interface that allows users to directly restore files from Dropbox without admin intervention.

Server Status

Once connected, Chronos will automatically report its current status to you. For example:

----------------------
SERVER
System: Ubuntu 22.04.5
Kernel: 6.8.0
Uptime: 6d|1h|2m
----------------------
MEMORY
Memory (RAM): 29GB/251GB [===>                              ] 11%
# = Compressed Memory
----------------------
DISK SPACE
User Storage: 9TB/20TB [===============>                  ] 46%
Env Storage: 1TB/10TB [>                                 ] 3%
OS Storage: 0.78TB/1TB [=========================>        ] 78%
SSD Health: 99%
----------------------
CPU
Average CPU Usage (5m): 0% [>                                 ] 0%
CPU Temperature: 35.4°C
----------------------
GPU
GPU Driver: 555.58.02 | CUDA Version: 12.5
GPU (VRAM): 248MB/49140MB [>                                 ] 0%
GPU Temperature: 32°C
----------------------
POWER
Main Power: Online
----------------------
CURRENTLY ACTIVE:
yonatan
yochay
merav
tim
----------------------------------
CHRONOS GUIDE: Link (Ctrl + Click)
----------------------------------

While we recommend that users inspect this to gauge the current workload on Chronos, and time their own work accordingly, this is only rarely needed. If, for example, many resources are coloured in red - the server is under heavy load and you may need to consult with the users currently logged in (shown in the status report) to know when you can use Chronos if you intend to run a heavy task.

At anytime, you can view updated status using the following command on any terminal:

status

Or, alternatively, any user can follow processes in real-time using:

htop

Running Tasks

Users work on Chronos in a multitude of ways, depending on the task at hand and the programming language being used. Sections specific to R, Python and Docker are found below.

An important feature of Chronos is the ability to leave a task running without the need to keep your connection open or laptop powered on. For RStudio users, this is managed automatically. For all other purposes, Chronos supports tmux (a terminal multiplexer), which allows the user to open any number of sub-terminals which remain active regardless of your connection to Chronos. For example:

tmux new -s ExampleSession #Create a new tmux session called 'ExampleSession'

From within this session, you can run tasks e.g. open a conda environment, initiate JuptyerLab and work in Python. The session will remain active, at all times, until closed by the user. To check your currently active sessions:

tmux ls

To exit a tmux session, without closing it, and re-enter it when needed:

tmux detach-client #From within an active tmux session to exit it
tmux a -t ExampleSession #From outside of a tmux session to re-enter a session

To permanently close a tmux session:

#From within an active tmux session:
exit

#From outside a tmux session:
tmux kill-session -t ExampleSession

Using R - RStudio

Chronos runs RStudio Server, which provides a desktop-like, graphical interface for R on Chronos. To access RStudio from your own machine, simply direct your web browser to:

192.114.18.118:8787

And login with your Chronos credentials. As with everything, the normal connection rules apply i.e. you must be connected to the Technion LAN or VPN to access RStudio.

This is a user-specific R environment. All packages installed are for you and you only - as are all RStudio settings and processes being run. You are responsible for updating packages and are free to uninstall/install packages at will.

RStudio Server provides a ‘Terminal’ tab which allows you to interact directly with Chronos as you would from any SSH-terminal. This allows you to, for example, change your password or check the current status of Chronos (see: Section: Passwords and Section: Status) .

To transfer files to and from Chronos using RStudio, use the ‘Files’ tab which provides ‘Upload’ and ‘Export’ (download) functions (found in ‘More’).

Important: To free resources for active users, idle R sessions - sessions that have not executed code for >=3 hours - are written to disk and removed from RAM. When restoring your session (which RStudio will do automatically upon launch), please allow 1-10 minutes for it to transfer data from the disk back into RAM ready for use - the interface will not load until this has completed.

If RStudio fails to reload after a prolonged period of time - which can happen when a large amount of data was left inside your R environment - you can clear your saved session by connecting to Chronos outside of RStudio and issuing the following command on any terminal:

clearR #Clear and delete all data stored in the R environment

To avoid this and any loss of work, we recommend including code to save RObjects (e.g. .RDS files) throughout your scripts whenever running a large job and terminating your scripts with code that clears the REnvironment automatically:

rm(list = ls()) #Clear R environment

No R session currently running code will be considered idle. In other words, you’re free to close your browser/laptop and Chronos will continue to execute your script to completion - no matter how long it takes i.e. there is no need for tmux when using RStudio on Chronos.

To end a session, click the red ‘off’ button on the top-right of RStudio. This will also wipe and remove your current environment from both RAM and the disk.

The default working directory has been set to your UserSpace. If this is not the case, or the working directory was changed, please reset it:

setwd('/media/chronos/Storage/[username]') #For example: setwd('/media/chronos/Storage/tim')

All system packages typically required by R in order to compile packages are already provided by Chronos. Should you encounter additional needs, please contact an administrator.

Finally, common R packages used in the lab (including their dependencies e.g. ggplot2) have been pre-installed for each user:

CRAN/Github Packages:
devtools
remotes
Seurat
BiocManager
stringr
msigdbr
glmnet
stabs
ggpubr
BayesPrism

Bioconductor Packages:
SingleR
celldex
ClusterProfiler
tradeSeq
glmGamPoi
CATALYST
diffcyt

Note: R packages requiring access to Docker/Singularity e.g. dynverse are not currently supported on Chronos for security reasons.

Note: Installed packages are not backed up and cannot be restored in the event of removal. Please re-install packages instead.

Using Python - Conda

Chronos supports Conda for the creation and management of self-contained Python environments. Conda is accessible from any Terminal connected to Chronos and all Conda installations are user-specific, and cannot be accessed, run or modified by others.

We assume Python users are already familiar with the use of Conda, however a starter guide is available here: Basic Guide

While a more advanced guide is available here: Detailed Guide

However, if you’re new to Python, please contact an administrator who can teach you all of the necessary details.

If you do not have Conda enabled for your user (i.e. the conda function is not available from a terminal) but you require it, please install it by following these steps:

#Change directory to your UserSpace:
cd ~/UserSpace

#Download the Conda installer:
curl -O https://repo.anaconda.com/archive/Anaconda3-2024.10-1-Linux-x86_64.sh

#Mark the installer as executable:
chmod u+x Anaconda3-2024.10-1-Linux-x86_64.sh

#Intiate Conda installation and setup:
./Anaconda3-2024.10-1-Linux-x86_64.sh

#Verify that Conda is now available to your user and delete the installer:
conda 
rm Anaconda3-2024.10-1-Linux-x86_64.sh

And follow the instructions, making sure to enable (‘Y’) conda init at the end of the process. Conda, as a software package, will be installed into your home directory e.g. /home/tim/Anaconda3. To update conda to the latest version:

conda update conda

It is now mandatory that all Conda environments are stored separate from your UserSpace. Chronos provides a dedicated 10TB HDD for this purpose, located at:

/media/chronos/Env_Storage/$USER #For example: /media/chronos/Env_Storage/tim

To create a new Conda environment called ‘EnvName’, at this location:

conda create --prefix /media/chronos/Env_Storage/$USER/EnvName python=3.11 #Python 3.11 is used for example purposes only
conda activate /media/chronos/Env_Storage/$USER/EnvName

All Conda environments are available from within any tmux session and they do not need to be explicitly installed within one. To list all available Conda environments:

conda env list

Since Conda environments often contain ~200,000+ files, backup is not feasible. For that reason, Env_Storage is excluded from Dropbox. Instead we encourage users to export a configuration file for each environment to their UserSpace (which is backed up). For example:

conda activate /media/chronos/Env_Storage/$USER/EnvName
conda list --explicit > /media/chronos/Storage/$USER/EnvName.yml

Which serves as a backup, and allows for easy and rapid recreation of the environment:

conda create --prefix /media/chronos/Env_Storage/$USER/EnvName_Recreated --file /media/chronos/Storage/$USER/EnvName.yml

Finally, to delete a Conda environment:

conda remove --prefix /media/chronos/Env_Storage/$USER/EnvName --all

Chronos contains a high-end, server-grade GPU to power Deep Learning and other AI-related processes through Python. To facilitate this, all Conda environments have automatic access to the GPU.

Using Python - JupyterLab

To provide a multi-tool, graphical interface for Python, we recommend installing JupyterLab into each, individual conda environment. For example:

conda activate /media/chronos/Env_Storage/$USER/EnvName
conda install jupyter

We typically recommend using JupyterLab, as opposed to Jupyter Notebook, as it provides additional functionalities including a ‘Terminal’ tab which allows you to interact directly with Chronos as you would from any SSH-terminal.

To access Jupyter, the service must first be mapped to a user-specific port. To check which ports are available and select one to use, see here: Chronos Port Mapping

Add this port to your installation by first creating a Jupyter configuration file:

conda activate /media/chronos/Env_Storage/$USER/EnvName
jupyter lab --generate-config #Replace 'lab' with 'notebook' if using jupyter notebook
nano ~/.jupyter/jupyter_lab_config.py

Adding the following lines to the top of the file:

c = get_config()
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = 5004 #Replace with your chosen port

And saving the file (ctrl + X, ‘Y’, ENTER).

JupyterLab can only access the files and folders stored within the directory it was launched from. For example, to launch JupyterLab with access to your entire UserSpace:

conda activate /media/chronos/Env_Storage/$USER/EnvName
cd ~/UserSpace
jupyter lab

To access Jupyter from your own machine, simply direct your web browser to:

192.114.18.118:[CHOSEN_PORT] #For example: 192.114.18.118:5004

Or use the desktop JuptyerLab application (https://github.com/jupyterlab/jupyterlab-desktop).

Upon first-time launch, an access token may be required. This can be obtained from the Terminal used to launch JupyterLab, for example:

http://localhost:5004/lab?token=43f93923c22222d8f0a0bdfd058f19203fe71d92590ef98a

Where the token is ‘43f93923c22222d8f0a0bdfd058f19203fe71d92590ef98a’. If this information is not shown on your terminal, press Shift + Q.

This configuration file and port will apply to Jupyter in all conda environments, however, if you wish to run multiple Jupyter instances simultaneously you will need to either generate separate configuration files (each with unique ports) for each instance or launch additional instances with a unique port directly:

jupyter lab --no-browser --port XXXX #For example: jupyter lab --no-browser --port 5005

Please ensure that you reserve any and all ports that you use in the linked spreadsheet above.

Alternative IDEs

While RStudio and JupyterLab are the recommended way to interact with R and Python on Chronos, any Integrated Development Environment (IDE) that supports remote access through SSH is supported. Among the many options available, we recommend using Visual Studio Code (https://code.visualstudio.com/) with the official ‘Remote - SSH’ extension to run R, Python and manage files on Chronos.

Using Docker - AppTainer

Chronos supports a multitude of image container formats, including Dockers, however, for security reasons, Docker itself is not available. Rather, Chronos pulls, manages and runs images using AppTainer (formely known as ‘Singularity’). This provides a rootless and secure way for each user to run images.

To enable AppTainer for your user, please install it by following these steps:

#Create a folder to install Apptainer into:
mkdir /media/chronos/Env_Storage/$USER/Apptainer

#Download and install Apptainer:
curl -s https://raw.githubusercontent.com/apptainer/apptainer/main/tools/install-unprivileged.sh | \
    bash -s - /media/chronos/Env_Storage/$USER/Apptainer

#Make apptainer available from anywhere on Chronos:
echo 'export PATH="/media/chronos/Env_Storage/$USER/Apptainer/bin:$PATH"' >> ~/.bashrc && source ~/.bashrc

#Verify that Apptainer is now available to your user:
apptainer

To pull and run, as an example, a Docker image:

#Create a directory to store containers:
mkdir /media/chronos/Env_Storage/$USER/Containers

#Pull and run a docker image:
apptainer pull /media/chronos/Env_Storage/$USER/Containers/hello-world.sif docker://hello-world
apptainer run /media/chronos/Env_Storage/$USER/Containers/hello-world.sif

In the above example, AppTainer pulls the ‘Hello World’ Docker image, converts it to .sif image (the native container format of AppTainer) and runs it.

Additional Services

Beyond providing platforms for R/Python/containers, Chronos will also provide a suite of additional tools in the near-future:

Chronos is able to run Large Language Models (LLMs, i.e. ChatGPT-like models) locally, as part of a service called ‘Bavel’ (as in Migdal Bavel).

Bavel not only provides a ChatGPT-like graphical interface accessible via any browser, but code-specific models will also be integrated into RStudio and JupyterLab to enhance troubleshooting efforts.

Bavel remains under active development and tutorials will soon be added to this guide.

Chronos runs Apache SuperSet - a database and live dashboard server - as part of a service called ‘Prometheus’.

Prometheus aims to provide users with a way to easily query databases (such as the upcoming IMMAGE/Ellison database) but also visualize, analyze and manipulate data in real-time all through an easy-to-use graphical interface accessible from any browser. Moreover, Prometheus allows users to build interactive dashboards to showcase their work to the team and to collaborators.

An example dashboard can be found here: Dashboard.

Prometheus remains under active development and tutorials will soon be added to this guide.

Finally, Chronos is flexible and should specific, new needs arise - these will be accommodated whenever possible.

Help & Support

If you have additional questions, require support or encounter any problems please do not hesitate to contact an administrator (Tim, Yochay).